Weak quantitative standards in linguistics research.

There are at least three types of unconscious cognitive biases [8,9] that can adversely affect the results of intuitive judgments, given the way that they are currently typically gathered in the syntax/ semantics literature: 1. Confirmation bias on the part of the researcher: researchers will often have a bias favoring the success of the predicted result, with the consequence that they will tend to treat data that do not support the hypothesis as flawed in some way (e.g. from a not quite native speaker, or from a speaker of a different dialect). 2. Confirmation bias on the part of the participants: individuals that the researcher asks to provide a judgment on a linguistic example – including the researcher him/herself – might be biased because they understand the hypotheses. When faced with complex materials, they could then use these hypotheses to arrive at the judgment. 3. Observer–expectancy effects (the ‘‘clever Hans’’ effect): individuals that the researcher asks to provide a judgment could be biased because they subconsciously want to please the researchA serious methodological weakness affecting much research in syntax and semantics within the field of linguistics is that the data presented as evidence are often not quantitative in nature. In particular, the prevalentmethod in these fields involves evaluating a single sentence/meaning pair, typically an acceptability judgment performed by just the author of the paper, possibly supplemented by an informal poll of colleagues. Although acceptability judgments are a good dependent measure of linguistic complexity (results from acceptability–judgment experiments are highly systematic across speakers and correlate with other dependent measures, but see Ref. [1]), using the researcher’s own judgment on a single item/pair of items as data sources does not support effective testing of scientific hypotheses for two critical reasons. First, as several researchers have noted [2–4], a difference observed between two sentences could be as a result of lexical properties of the materials rather than syntactic or semantic properties [5,6]. Multiple instances of the relevant construction are needed to evaluate whether an observed effect generalizes across different sets of lexical items [7]. The focus of this letter, however, is on a second problem with standard linguistic methodology: because of cognitive biases on the part of the researcher, the judgments of the researcher and his/her colleagues cannot be trusted (Box 1) [8,9]. As a consequence of these problems, multiple items and multiple naı̈ve experimental participants should be evaluated in testing research questions in syntax/semantics, which therefore require the use of quantitative analysis methods. The lack of validity of the standard linguistic methodology has led to many cases in the literature where questionable judgments have led to incorrect generalizations and unsound theorizing, especially in examples involving multiple clauses, where the judgments can be more subtle and possibly more susceptible to cognitive biases. As one example, a well-cited observation in the syntax literature is that an object–subject–verb question asking about three elements is more natural than one asking about only two (e.g. What did who buy where? is claimed to sound better than What did who buy?). Several theories explain and build on this purported phenomenon [10,11]. However, it turns out that the empirical claim is not supported by quantitative measurements [12,13]. There are many other such examples of questionable judgments leading to unsound theorizing {[2–4]; including an example from the first author’s PhD thesis (Gibson, E., 1991, PhD Thesis,